There appears to be a bug in the ncio library on Hercules resulting in an HDF5 error when attempting to create a dataset. A bug report has been written in JCSDA/spack-stack#916 with more details.

It appears that this is a problem with netCDF4 and not the NCIO library after all. The UFS model folks are reporting a very similar issue with their RTs (see ufs-community/ufs-weather-model#2015). @DusanJovic-NOAA just noted that the I_MPI_EXTRA_FILESYSTEM=ON may fix the issue. I tried this out on my test case and it worked 10 out of 10 times, so I think we have a pathway forward! I will add this variable to the global-workflow/envs/HERCULES.env for the fcst, efcs, and analcalc jobs.

This ran for a total of 46 cycles over C96/C48 and C384/C192 resolution tests without a single HDF5 error, or any other issue, so I believe this is ready to go.

This was tested on Hera as well over 5 cycles and ran successfully each time.

On a different note, the CI sp version (2.5.0) is newer than the one used to compile/run on systems (2.3.3). When version 2.3.3 is requested, both 2.3.3 and 2.5.0 are concretized by spack (I believe this is because ip requires sp and defaults to the newest version, which is 2.5.0). This causes CI to fail when attempting to build. This was encountered during the UFS_Utils upgrade where @AlexanderRichert-NOAA made the suggestion to change to 2.5.0.

@RussTreadon-NOAA
If there is no objection, we would like to merge this so we can continue Hercules porting for the cycled system.

If @DavidHuber-NOAA is satisfied, that's enough for me. I do not monitor this repo.